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what's this? 
Hil This is me: 
Sura Evans 


blog: JVOS CH & 
twitter: Ob0rk 


and in this zine I want to tell you about 









how I got 
better at 
debugging 


These are 5 Ways Ine changed how I think 


about debugging, 
y Remember the bug is happening 


for a logical reason. 


Tt's Never magic- Really. Even when it makes no sense. 


E] 8e confi oon L can fix it 


5 this Cowell Tve fixed 
before: E alot of hard 
bugs before 


4 Talk to my 0 


gë 78 





Y know my ORE S toolkit 
etore: 

T want to know 

STHING but I 

don't Know how 

to Sind out 


now: 













Y most importantly : T learned to like it 


Yoh no ey I think T7 
akg Y n T m 









about to learn 
Some thing 


x . 
ds facial ex pression: 
determination 


T can't teach you in 20 pages to Y debugging 
(though I'll try anyway!) T can show you some of 


my debugging toolkit} though! 


These are the tools L reach for when I have 
a question about a program I want to Know 
the answer to. By the end of this, T hope 
+o have given you a few new tools to use! 


Section 1: I/O and 
X% System calls + 


Hello, dear reader) In this zine, there 
are 3 sections of tools that I love. 


For each tool, T Il tell you why it's useful 


and give an example. Each one is either 


ES 


Some of the most basic questions you 
miaht have when you log into a 
mus behaving machine are: 


- is this machine writing to or reading 
from disk? The network? 


- are the programs reading files? Which 
files? 
So, were starting with Finding out 


which resources are being used and 
what our programs are doina Lets go 7 


NS 
AE 


= 

ae a 

dstat [= 
L love dstat because Es 


super simple : Every second, it 
prints out how much network and 
disk your computer used that second. 


i 


Once Lhad an intermittently slow database 
server. I opened up dstot and stared at 
the out put while mon itoring database speed. 


send |recv 
\ 

A during this period, 

Sk everything is normal 


43 MB GETS SLOW 


6 3 back to normal 





( 

JJ 

i] 
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l 300MB1 DATARASE 
i 

t 

i 


Could 2GoMB coming in over the 
network mean... a ZOOMB database query?! 
>yEesvi 
This Was an AWESOME CLUE that 
helped US isolate +he problem query, 


Ñ iy 


o 


O 
RTS 
Strace Ls my favourite program. Tt prints 


eS system call your program used. T+'s 


a cool way to gef an overall picture of what your 


program Ts doing, and L Y using itto answer 


questions like “which files are being dd : 


C 
E strace python my_program. py | | ka 


reos TO BOREL config_file") = 
get read(3, "the contents of the fete") 
PS hundreds of lines .. 
$ (comect(s, "172.217.0.163") 
& lsendto(5, “hi!!") 
WARNING 
strace can make your pragram 


run SOx slower. Don't run it on your 
production database 


L cant do justice to strace here, but I have 
A whole other zine aboutit at 


jvns.ca/zines 


opensnoop Y, 


eBPFY 


(kind of) 


When you run 


lo pensnoop -p §PID | 


EPA EA: 


it will print out every file 


being opened bu a program. You might think 


a0? Strace can do this 
ľ tool Tust use 


strace -eopen -p $PID 







and you would be right. But 
Strace can make your 
program run lOx slower. 


Open snoop won't slow you down 
= how to get it Z 
Requires : Ubuntu 16.04 + 


ora ~44+ Kernel version 


= how it works = 

opensnoop 'S A script 

that uses a new kernel 

Tnstallation instructions at: en ar Jeers 
eB 

github. com / iovisor (bec ni 


There's also an opensnoap 
on OSX gsol 

That one is powered 
by DTrace. 







Shere are. lots of eBPF- 
powered tools! Check out 
that GitHub repo to 


\earn more: 









SSS eb lol lr 


section ¿e yn etworking E 


” 
NE TE Git 


Tve devoted a lot of space in this zine to 
networking tools, and I want to explain why. 


A lot of the programs T work with 
communicate over HTTP. 
response 


request > 
: GE T =7 v =7{ name : “Frofru ia 
/co+s/42." program colour : “blue | 


Every programming language uses the 
some network protocols Y So the network 
iS a nice language -inde pendent place ta 
Answer questions like: 


* Was the request wrong , or was 
it the response! 
* Ss my service even running ? 


* my program is slow. Whose fault 


is that ? 
Let 's go Y 


Ma qxe = 

neta ce oon 
OS ON ` 

OW 4, Seca! 


HTTP request S are Fundamentally, 
really simple — they re gust text! To see 
that, let's moke one by handY First, make a file: 


— = — 


1 
| request. txt | 
an eee \ 


| GET / MTtP/1.1 
| Host: ask.metofilter.com ı 
i User-Agent . zine 1 

4 
y I 
i 
{ 


(2 new \ines! important |!) 


A ee ea! 


Then: 7 nc stands for netcat 


Beh A er 


Sp ce aa v \ 
¡$ cat request. txt | nc metafilter. com 80 | 


You should get a response back with a. bunch 
etcat to send 


of HT™L | You can also use n 
huge files over a local network quickly: 


step1 : (on target machine) step2: Lon the sou rce) 


TEE | | 
1! cat bigf ile | \ 
' nc 192.168. 2.132 981] 


192.163. 2.132 --- 
l 
nc -A 9931 7 bigfile | eee aa 


$ 
this sends the data ) 


“this listens on the port l 


lx 
& netsta 


receive a request, a program (aka"server’) 
needs to be" listening bathe port. Finding 

out which programs are listening on which 
ports is really easy . Its just 


Ever network request gets sent ta 
a port (like 80) on a computer. To 


xo 
yo go 


a n 
dk tuna, please Y K NOE so 


_ also known as ou wk OF 
(sudo netstat Stone i 
Here's what youll seez SR 
proto \ocal address PID / program name 
tcp 0.0.0.0 +5353 2993 / python 


port 
So! TL Y netstat because 
it tells me which processes 
are cunning on which ports. 


On OS X , use | Isof siecle instead. 


== oo 





narep iS my favourite starter network 
SPY tool $ Try it right now! Run: 


| tafilter | 
sudo NACER -d ong e a er; 


` 


Then go to http: // metafilter. com 
in your browser. You should see 
match ing network packets in ngreps 
output Y We are SPIES Y 


Recently at work T'Y made a chang e 
to a client so that it sent 


{“some-id ">. with all its 
requests. T wanted ta make sure 


iF was working, so T ran: 
ARMA ee 
; Sudo ngrep some_id 7 


Y 


a= 


“eee — e. — — —— 


TÍ found out that everything was ok vb 


pop tcpdump Ù 


cpdump is the Most difficott 
networking tool we'll discuss here 
and it took me a while to y ct. 


T use it to save network traffic to 
analyze later | 











See 
jvns.ca/zines 
For a zine 
all about 













“port 8997" is 
actually ating 


Sudo tepdump port 807 





\ 
— program inthe 
1 TW Service. pcap 1 | “Berkeley Packet 
il ilee" BPF) 
J language . BPF 
a `pcap fi le” (© packet capture”) is the a get compiled 
standacd for saving network traffic. hag re oe 


Everything understands pap Y 


Some situations where TIL use tcpdump: 


is Tm sending a request To A machine and 
I want to know whether it's even getting there, 
Q _tepdump port 80; will print every packet on port ga) 
* T have some slow network connections and 
T want +o Know whether to blame the cl ient 
or server. (we'll also need wireshark!) 
* T just want to print aut packets to 


See them (tcpdump -A) 


wireshark 


Wireshark is an GUL tool 


for network analysis. Here's an exercise ta 
Learn ¡AY Ron this: 


¡sudo tcpdump port 80 -w http.peap | 
While that's running, open metafilter. com in 
your browser. Then press Ctrl+C to stop tcpdump. 


Now we havea pcap Fileto analyze! 


¡wiceshark http -pcap | 


Explore the Wireshark interface | 
Questions You can try to answer > 


O What HTTP headers did your browser 


send to metafilter.cam? 
Chint : search Frame contains “GET"!) 


D How long did the longest request take? 
(hint: click Statistics > Contereutiens ) 
®© How many packets were exchanged 


1 ipf 
with metafilter. com's servers AA 
(hint: search {ip dst == S4.1g6.13.33 | Netan econ 


section 3: CPU + 


Your programs spend a lot of time 
on the CPL! Billions oF cycles. 
Whot are they DOING 21 


This section is about using spect 3 
to answer that question. perf is a 


Linux ae tool that is extremely 
useful and not as well -known as 
it should be. 

Cin eneral, muy aim in this zine is to showcase 


tools that I think dont get enough love Q) 


Some things I didn't have space for in this 
section but wanted +o mention anywa g a 


svalar ind 
«the Sava ecosystem's fantastic 
tools (3stack, Visual VM, Youckit) 
which your language is probably jealous of 
* ftrace (For linux Kernel tracing) 
* LTT (ditto) 


+ eBPF 


Q perf y 


perfis not simple or elegant. Tt is a weird 
mult itool that does a few different , very 


useful things. First, it's a A sampling 7_ 
=Xprofilep/ = 


Try running : 


$ sudo perf RWE python de 


(press CAC after. a few Second s} 





You can look at the results with: 


ig Sudo perf report 
Mine says it spent 5% of its time in the 
PyDict- Get Item Function. Cool! We learned 
a tiny thing about +he CPythen interpreter. 


if you use perf to profile perf can be installed an 
a Python program, it'll pretty much any Linux 
Show you the C functions machine. The exact 
ae from the CPython Features it has will 
interpreter, not the rer ee j 

Py Mon functions . P see 


Kernel version. 


pect iS for everyone 


One daw, , I had a server that was USING 
100% of its CPU.Within about 60 seconds, 
Í knew it was doing regular expression 
Matching, in Ruby. How? ‘pect top’ is like 
top, but for Functions instead of programs . 


E sudo pe + to pr 
i process PID Zo function 


i ruby 19587. FE match at 


perf top doesn't always help. a Ruby's internal regexp y 
But it's easy to try, and sometimes 7, matching function , 
T learn something Y ~ 





se especially Sava and node devs Y 


Remember when T said perf only know S 
C fonctions? THs not quite true. node.js 


and the SVM (java, scala, clojuce...) have both 
taught pec about their functions. 


= Z : | Java | Z 
Use the Look up ‘perf -map -agent 
-- pecf-basic-prof on GitHub and follow 
Commard line. option the directions 


eRrkhh + 
Flame caps are an Awesome way to 


visualize CPU performance, popularized by 

Brendan Gregg's Flameg raph -pl tool. F 
E aithub.com /brendangegg / flame graph = 
Y 


NY 
Here's what they lock like: 






[bre 20% ] [teeth 28% 


panda 20% | alligator 80% 
main 100%, 


Theyre constructed fram collections (osvallyy 
thousands) of stack traces sampled from 

Q pogam. The one above means 307, of 
the stack traces started with 5 


And 10% with “ Main , 


main 
aligator 
Panda 


eat 
You can construct them from ‘perf ‘recording 5 


(see Brendan Gregg's Flame graph github for how) 


but lets of other unrelated tools can produce 
them too. T Y them. 


spy on your CPUY 











tip , 
Google “Latency 
Numbers every 
programmer 

Should Know Y 


Your CPU has a small 
cache on it (he LL cache \ 
that i+ can access in 


=0.S nanoseconds | 
faster than RAM \ 
TF you're trying to do an operation 


in microseconds, CPU cache usage matters | 


‘tho 


fa AA LR pass -e € 
A how to} | perf stat Is; request as 
Y use it A AS pecifre Statistic 
This runs 'As' and primis a cepoct at the end. 


f how it ] Your CPU can track all Kinds of counters 
A works | about what it's doing. perf stat! asks 


it +o Count things (like LL cache 
£ cepoct the resolts. 










program is USING 
se. caches ? 





L1 cache misses) 


Hardware is cool. Te never used perf stat 
iN earnest but I think it's awesome you 
can get So much info From your CPU. 


T hope you learned 
Something new. 
Thanks for reading Y 


Thanks to my partner Kamal for 
help reviewing and ta the amazin 
Monica Dinculescu (QA not waldorf) 
for the cover art. 

To learn more, see: 


k*k my blog > JVNS.CaA 
X my other zines: Jvns.cal/zines 
x brendangregg. com 


But really YOU just need to experiment. 
Try these tools everywhere. See where they 


help you track down bugs and where they don”. 


o$ Stace really 
helped with 
that problem Y 


Tt takes practice, but L Find these tools 


both fun and a useful Job skill. T hope 
you will too Y 









that didn'*+ 
tell me much, 
oh well? 









like this ? 
there are more 
Zines at: 
http://jvns.ca/zines 
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